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Bad software can tarnish the brand...or kill the company 
Knight Capital sada cris! a Cost It $440 Million 


By NATHANIEL POPPER AUGUS 


Runaway Trades Spread Turmoil Across Wall St. 





is 6; i et The scandal cost Martin V iial-skoa his position as chief executive of WW 
Errant trades from the Knight Capital Group began hitting 
tha New York: Stock Fachan ge almost as soon as the https://goo.gl/T96ezC 


“riaminmer he = cr Vil ns yor air 
opening bell rang on Wednesday. 


onl ary: a nous 9 
The Knight Capital Group announced on Thursday that it lost $440 Wi l Diesel Gate KI l VW , 


million when it sold all the stocks it accidentally bought Wednesday 


morning because a computer glitch. https://goo.gl/7dHOjO © 2021 Philip Koopman 2 
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= Software quality problems are pervasive OW Chrysler recalls 

e Are you going to wait until you're on 1.4 million Nackable ee 
CNN to do something about it? . 7 by David Goldman (© July 24, a \\\ 
= Your company lives or dies by its > So 
software quality S| * YY 
e Software is a core competency ... ] 7 | 

.. Whether you like it or not CHRYSLER 








5 : | be 0 01:23 / 02:08 
e Embedded software requires unique Skills & Chrysler is recalling 1.4 million vehicles that can be remotely 
technical approaches hacked over the Internet. https://goo.gl/97fY8H 
= More product-level testing wont make this problem go away 
e Need good practices, development process, development skills 


=m Get serious about software quality 
e Daily practices, process support, training, metrics 
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Embedded Software Is Challenging Melon 
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= Customers expect “perfect” embedded SW MyFord Touch problems: Ford to issue 
a upgrade 
e Everyday desktop quality software isnt good enough 
e Bugs can lead to class action lawsuits 
e Upgrades can be painful to deploy 





Glitches in MyFord Touch software that replaces knobs and buttons witha 
touchscreen have led to plummeting user approval ratings for Ford cars 


= Significant technical challenges 
e Limited hardware resources 
e Real-time operation 


e Interaction with system-specific 
sensors and actuators 





©} MyFord Touch has been plaqued with software problems PR 
m Most embedded software is Mission Critical Charles Arthur and agencies wondsy7 November2011 02.51 £5 
The motor company Ford has discovered belatedly that touchscreens don't make 
® Safety - someone g ets ki | led or i nj U red a great replacement for the knobs and buttons of a dashboard - especially if the 
touchscreens are plagued with software glitches. 
6 M iss i on & ritica | : fai | U re res U lts ; Nn The company says it will send memory sticks to 250,000 customers in the US 
° offering a software upgrades for its glitch-prone MyFord Touch system, which 
unacce pta b | e | OSS (m on ey, b USINeES Ss; ae .) replaces the standard dashboard knobs and buttons with a touchscreen. 
https://goo.gl/4bS5rd 
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3. Software assembly for power train ECU TOY-MDL04983210 
After the 4" Steering Committee, rebuilding of engine contro! and actions for software assembly were 
started. 
(1) Achievements 
\D Identification of current issues with software assembly ..... Ongoing 


{Control structure reform has already started in Engine Div. In coordination with this, 
software structure retorm will be carried out. Asa first step, it has been decided to transfer 
two employees from Engine Div. and carry out trial with purge control.) 





both 
: TMC and suppliers struggle to confirm 
| overall situation 


(0 0s | | 0 SB | | 


23 TOY-MDL04983219 


https://goo.gl/v8CY62 TOY-MDL04983253 











Toyota's killer firmware: Bad design and 
its consequences https://goo.gl/pX3qgb 


Michael Dunn -October 28, 2013 


On Thursday October 24, 2013, an Oklahoma court ruled against Toyota in a case of unintended 
acceleration that lead to the death of one the occupants. Central to the trial was the Engine 
Control Module’s (ECM) firmware. 


© Toyota’s electronic throttle control system (ETCS) source code is of unreasonable quality. 

e Toyota’s source code is defective and contains bugs, including bugs that can cause 
unintended acceleration (UA). 

© Code-quality metrics predict presence of additional bugs. 

e Toyota's fail safes are defective and inadequate (referring to them as a “house of cards” safety 
architecture). 

e Misbehaviors of Toyota’s ETCS are a cause of UA. 


Toyota Says It's Settled 338 Cases So 
Far In Acceleration MDL ttps://g00.g/BL95KF 


By Aebra Coe 


Law360, New York (July 22, 2015, 11:37 AM ET) -- Attorneys on both sides of 
multidistrict litigation over deaths and injuries caused by alleged unintended 
acceleration in Toyota Motor Corp. vehicles told a California federal judge on 
Tuesday that the settlement process continues to hum along, with deals reached in 
338 cases, up from 289 in March. 
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This is the bad line 
of code for 
Heartbleed: 


memcpy (bp, pl,payload) ; 


e Classic buffer overflow 
vulnerability 
— Copies “payload” bytes 
from pl to bp 
— Reads other user's data, 
including secret keys, 
if payload value is too big 
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How Heartbleed Works: The Code Behind the 
Internet's Security Nightmare 


https://goo.gl/1Joxy2 





By now you ve surely heard of Heartbleed, 


that exposed countless encrypted transactions to any attacker who knew how 
to abuse it. But how did it actually work? Once you break it down, it's actually 
incredibly simple. And a little hilarious. But mostly terrifying. 
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Large Scale Production = Big Problems 





Nest Learning Thermostats in the UK fail to 
spring forward to British Summer Time 


ti) Toe 





https://goo.gl/C9775V 


Honda, Yes Honda, Recalls 175,000 Cars For 
Unintended Acceleration =" 





Bloomberg reports that all new hybrid Honda Fit subcompact and Vezel small 


crossover models sold in Japan since last will be recalled due to a software 
problem with the engine control system. They did not elaborate, but said the 
problem could lead to unintended acceleration. https:/ / goo.gl/ Hrr7ci 


thermostat bug plunges 
customers into cold 


By James Billington https://goo.gl/RPv9V6 
January 14, 2016 14:27 GMT 





Smart thermostat has been leaving customers cold after suffering 
from a software bug that drained its battery. 





theguardian 


Samsung keyboard bug leaves 600m 
Android devices exposed to hackers 


Vulnerability remains months after discovery, allowing hackers to eavesdrop on 
calls, steal data and activate camera, microphone and GPS remotely 








© Hundreds of millions of Samsung smartphones are vulnerable to hacking thanks to the built 


Photograph: Samuel Gibbs fo Se tie Gard an https: //G00. gl/FYW7ZH 
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There Are Too Many Examples 





= A steady stream of software 
mishaps, recalls, etc. 


Y GREENBERG SECUR 


HACKERS REMOTELY KILL A JEEP ON THE 
HIGHWAY WITH MEIN TT 
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Miller attempts to rescue the Jeep after its brakes were remotely disabled, sending it into a ditch. (0) anoy 


GREENBERG/WIRED https://goo.gl/o2FuqZ 


Airbus confirms software configuration 
error caused plane crash 


Airbus A400M flight recorder data confirms "quality issue" in setup caused failure. 
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This Goes Far Beyond Transportation 
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HACKER CAN SEND FATAL DOSE. TO 
HOSPTTNE DRUG PLP 


https://goo.gl/I6QLEK 
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Hospira's drug infusion pumps include a serial cable (the wide grayish-white cable with the single red stripe on one 
edge) that connects the communications module to the main pump board. (9) BILLY RI0s 


administer. Because the libraries don’t 
require authentication, Rios found that 
anyone on the hospital’s network— 
including patients in the hospital ora 
hacker accessing the pumps over the 
Internet—can load a new drug library 
that alters the limits for a drug. 








Crypto weakness in smart LED lightbulbs 
exposes WI1-F1 passwords 


More evidence the Internet of things treats security as an afterthought 


by Dan Goodin - Jul 7, 2014 3:20pm EDT 


https://goo.gl/tHGiAO 


6LOWPAN mesh network 





In the latest cautionary tale involving the so-called Internet of things, white-hat hackers have devised 


an attack against network-connected lightbulbs that exposes Wi-Fi passwords to anyone in proximity 
to one of the LED devices 


The attack works against LIFX smart lightbulbs, which can be turned on and off and adjusted using 
iOS- and Android-based devices 
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Act As If Your Products Live Or 
Die By Their Software 


BOM = Bill Of Materials 


Software 
0% of BOM cost 
90% of product 


differentiation Mechanical System 


90% of BOM cost 
Mostly commodity 





Electronic Controller 
10% of BOM cost 
Mostly commodity 
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Product Testing Wont Find All Bugs beg 
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= Testing bad software simply One third of faults take more 


makes it less bad than 5000 years to manifest 
e Testing cannot produce good exe" bi Juma Aesearcnand Development 
software all on its own 28(1), p. 2-14, 1984. (Table 2, pg. 9, 60 kmonth column) 


e Your customers will regularly 
experience bugs that you will 
not see during testing 






ye e For most products, you cant 
coe? 
< 


even test 5000 years 


OPERATIONAL 
SCENARIOS 





TIMING AND SEQUENCING 
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_ How Bad Can It Possibly Be? as 


= For YOUR product, what is the worst possible outcome: 


e For a software bug? ~ ie ' 
— People killed or injured? | : | 
i) * Vin) ]i ih dinug | 








— Property damage? 

— Cost to deploy a fix? 

— Loss of brand reputation? 
e Fora malicious attack? ” 
e Hint: 7he answer Is the same oO 

for both bugs and successful attacks 

= Regulation is likely to increase 

e IEC 60730 safety standard required for European appliances 


e Security standards are already proliferating 
© 2021 Philip Koopman 12 


YOU WILL BE KILLED 
BY ROBOTS 





https://goo.gl/OSfG8i 


| Designing For Safety Nelo 
m= Every system is assumed to be unsafe by default 


e It is up to you to proactively show that it is safe 
» Example: DEF STAN 00-55 Parts 1 & 2 


1. Collect risks 
e What can go wrong? What does “safe” really mean? 


2. Assign risk severity 

e What types of mishaps are most important to avoid? 
3. Perform risk mitigation 

e How can you avoid hazards and activation of hazards? 
4. Develop software to acceptable level of integrity 

e Ensure that risk mitigation is successful 
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Risk Identification & Assessment Mellon 


| : University 
= Create a Hazard Log (list of hazards), including HAZOP 
= PHA (Preliminary Hazard Analysis) & Risk Table 


Probability 




















e E.g. Consequence Le = Par wleaeeaidea | oe Ve 
- $100M loss RISK - : Ww 


High Low 
— S1M loss 


Very Very Very 
7. ? oe High (4) High (4) High (4) 
e E.g. Probability ILE (2) (2) 


quence 


















=F (2) (2) 
— Every 10 years 


e (4) .. (0) & See SIL on next slide 
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nllgba. Sf>-lnyoweg Ext ipige 16 RAGOL gS he 


m SIL = Safety 
ie 1 Faultdetection and diagnosis | 8.1 | OE RO GR HRD 
I r 
Integrity a Se 
Failure assertion programming ie Be 
Level 
a ee ne ee 
‘ mai er [8 _Bverse programming | a | OR | OR | OR OCR 
‘ SMO ISS [a Recoverybiock TR TR TR TOR 
e Usedtodetermine = [se _Backwardrecovey | 7 | OR OT OR | OR | OR 
Pepa Ol sf_Forwardrecovery | @ | OR | OR | OR | OR 
ite lS ol [sq_Restry fault recovery mechanisms | 3.9 | OR | RJR (HR N 
sh_Memorisingexecutedcases | 8.10 | ER | RD KR 
= Example: 4 Gracefuldegradation tt | OR | OR ERE 
IEC 61508 CL 
© HR= Highly [6__Dynamic reconfiguration | 8-13 | 


7a Structured methods including for example, JSD, 
Recommended MASCOT, SADT and Yourdon. 


OO a A NS 
B.7 
e NR=Not — 
Formal methods including for example, CCS, CSP, HOL, 
Recommended ea a 


(don't do this) [8 Computer-aided specification tools lec en508)_ | 8.24 | oR | R_ [CAR | HRD 
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_Head Count: Half Designers, Half Testers Umer 


Validated software 





System level ! Soltware level 
requirements [| > requirements | 
l specication | specification 











Validation 
BEEBE BBR ERE RRR 







Verification 


| T | Test document 


Design document: 
| S| specification 








Figure H.1 — V-Model for the software life cycle “—— 


fEC 2570/73 


IEC 60730 Appliance Safety [IEC 60730] © 2021 Philip Koopman 16 
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Essential Practice: Peer Reviews a 


=m Gold Standard: Fagan Style Inspection 
e Pre-review meeting 

Formal meeting 

Written review report 

Follow-up and possible re-inspection 


The more formal the review, the higher the payoff 


m Good reviews find 50%+ of defects for about 10% of project cost 
e Defects are found early, when they are cheaper to fix and cause less disruption 
e Why is it so many designers say they dont have time to do peer reviews? 








m Other technical issues are crucial for good embedded software 


e Watchdog timers, mutexes, Rate Monotonic Scheduling, interrupts, exception handling, 
reducing code complexity, secure update, timekeeping, performance optimization, ... 
© 2021 Philip Koopman 17 


Security Matters for Industrial Systems! 


Hack attack causes ‘massive damage’ at steel works 











= Attacks can affect the physical world 


https://goo.gl/CDsbV2 


Attacks Against SCADA Systems Doubled in2014: Dell 


By Mike Lennon on April 13, 2015 
Dell SonicWALL saw global SCADA attacks increase against its customer base from 91,676 in 
January 2012 to 163,228 in January 2013, and 675,186 in January 2014. 
https://goo.gl/24Jp7j 
Key SCADA Attack Methods 


Source: 2015 Dell Annual Security Report 


22 December 2014 


The hack attack led to failures in plant equipment and forced the fast shut down of a furnace 


A blast furnace at a German steel mill suffered "massive damage” following a cybe 
attack on the plant's network, says a report. 








—— ess a2 ae. 
https://goo.gl/rYgWFf —r lee Sele Neb Pace Generation (Ci te Scripting) ee 


Ukraine has been forced to turn to back-up power sources in recent months following a spate of power cuts 

A power cut in western Ukraine last month was caused by a type of hacking known as are 

id a Apis . © 2021 Philip Koopman 1 8 
spear-phishing", says the US Department of Homeland Security (DHS). 
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The Bad ihe are Atta more than credit card numbers 


A hyd roelectric pla Nt — TWreine De BRUAUX 
French electric companies apparently | Semen 
Puss totale Niveau 


like to put their hydroelectric plants <7 kwh) jamont 











online. Tentler found three of them 58,7 kw) [260.3 Imm 
using Shodan. 
TIMARCHE 
This one has a big fat button that lets —s 
you shut off a turbine. But what's — 
2 } 58,700 Watts between friends, right? nme 
Lilien ignt controls It's not just France that has a problem. _ aes 
IMAGES PRECEDENTE 


The U.S. Department of Homeland 
When something that literally anyone — ; 
Security commissioned researchers Wait, does that say kilowatts? 
in the world can access says "DEATH 
MAY OCCUR !!!" it's generally a good 


idea to build some kind of security 


last year to see if they could find 
using Shodan. They found several. 


Tentler told DHS about all the power plants he found -- actually, DHS called him after he 


accessed one of their control systems. https://goo.gl/tPrcB6 


“a big fat button lets you shut off a turbine” 
(No login credentials required) 


- 
s =s 


around it. 
Oops - no. For some reason, someone 


a, 
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mee 


thought it would be a good idea to put 
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traffic light controls on the Internet. 
Making matters way, way worse is that 


these controls require no login 





credentials whatsoever. Just type in 
the address, and you've got access. ila ile acai © 2021 Philip Koopman 19 
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Designing For Security Mellon 





=m Security testing isnt enough 

e Bad code is especially vulnerable 

e Testing mostly finds known problems 
m Need to address: 

e Security requirements 

e Characterize threats & risks 

e Security risk management plan 

e Deploying security patches 
m Myriad technical issues 

e Secure update, cryptography, input 

validation, least privilege, code quality, 


passwords, privacy, web interface, error 
handling, secure coding, ... 





_ University 


Forbes ms https://goo.gl/FQ4jen 
Shopping For Zero- -Days: 

A Price List For Hackers' Secret 
Software Exploits  *™Sreents 


ADOBE READER $5,000-$30,000 
MAC OSX $20,000-$50,000 
ANDROID $50,000-$60,000 
FLASH OR JAVA BROWSER PLUG-INS $40,000-$100,000 
MICROSOFT WORD $50,000-$100,000 
WINDOWS $60,000-$120,000 
FIREFOX OR SAFARI $60,000-3150,000 
CHROME OR INTERNET EXPLORER $80,000-$200,000 
10S $100,000-$250,000 
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Testing Alone Won't Fix Bad Software filer 


= You cant test in quality, safety, or Poet 


= In an ideal world, ' 
throw it away and startover ff . P | 
e But, the world is not ideal ... a a ae 
= Incremental Reengineering \ i teres es a Hi i writ | 
e Identify & fix high risk modules NY ey as 4 RN 
e Clean sheet for each module; don't try to derive design from code 
= Improvement requires cultural change 
e Requires commitment to good software at all levels of organization 


e Commitment must survive a “but we have to ship next week’ crisis 
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Top 10 Embedded SW Warning Signs fetes 


University 


Software time estimates are driven by external dates 
Process steps skipped during schedule crunches 
Software development is simply “coding” plus “testing” 
Poor traceability from product test to requirements 
Bugs due to poor code style & complexity 
Bugs in software fault detection/recovery 

No Security Plan; no Safety Plan 
Tester:Developer ratio too far from about 1 : 1 
More than about 5-10% of bugs are found in product test 





0. Fewer than 50% of defects are found by peer review 
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The Path To Good Software 


CAPABLE PEOPLE SS 


BAKED-IN 
SOFTWARE 
QUALITY 


ROBUST PROCESS —> == 


EerRAnTiCEes, 
TECHNOLOGY 
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_ Software Quality, Safety & Security pa 


=m Software is crucial for providing value 
e But — even a single line of bad code can kill a product (or a company) 
e Writing software is a high-stakes profession. Take it seriously. 
= Good software requires process + technology + people 
e Embedded software requires unique technical approaches 
e You cant test quality, safety, or security into software 
= Good process enables good software 
e Whether “V" or agile, need to actually follow a good process 
e Typically need 1:1 head count for testers:developers 
e Peer reviews find 50%+ of defects on the cheap — why aren't you doing them? 
m= Safety and security are essential — don't wait until there is a loss event 
e Most embedded software is safety critical or mission critical 
e Security is required in essentially all embedded software 
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What Happens Next? 


m Assess where you are 
e How good is your code quality? 
e How good are your software, process & technical skills? 
e How good are your safety & security practices? 

= Improve process, skills, technology 
e Ensure you are doing effective peer reviews | 
e Formalize and follow a reasonable software process 
e Adopt/adapt relevant safety & security standards 
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e Ensure developers have strong embedded software & process skills 


= Cultural change 


e Make software quality a first class company goal, not a sideline 


e Daily practices, process support, training, metrics 
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